Multi-Screen Interaction

Definition and Technical Concept

Multi-screen interaction refers to a technical ecosystem where displays across various positions in the cabin (including the digital instrument cluster, central infotainment touchscreen, front passenger entertainment screen, rear roof-mounted display, armrest control screen, HUD, etc.) achieve seamless data flow, collaborative operation, and personalised shared experiences via a unified software architecture and communication protocols. This technology dismantles the traditional "information silos" where cabin screens functioned in isolation, turning all in-vehicle displays into a unified, collaborative interactive space.

Technical Implementation Architecture

The technical implementation of multi-screen interaction relies on the coordination of three architectural layers. At the hardware level, it adopts a domain-centralised architecture powered by a single high-performance cockpit SoC to drive multiple displays simultaneously, replacing conventional distributed setups that required discrete controllers for each screen. At the system level, built on automotive operating systems such as Android Automotive OS and HarmonyOS Cockpit, hypervisors are deployed to achieve multi-OS isolation and unified resource scheduling. At the communication level, standard protocols like Wi-Fi Direct, DLNA, and Miracast—along with proprietary wireless transmission solutions—enable low-latency content streaming and screen synchronisation across displays.

Core Interaction Scenarios

Multi-screen interaction spans multiple dimensions, from content sharing to collaborative controls. In navigation relay scenarios, front or rear passengers can input a destination on their respective displays and push it to the central infotainment screen, which then synchronises the route with the digital instrument cluster and HUD. In media sharing scenarios, active music or video playback can seamlessly transition from the central screen to the rear displays, enabling seamless "cross-screen handover". In terms of ecosystem-wide continuity, operating systems like HarmonyOS Cockpit support cross-device transfer for navigation, audio, and video conferencing, allowing location addresses on a smartphone to be sent to the head unit navigation with a single tap.

On the functional collaboration front, rear passengers can adjust front multimedia playback, manage air-conditioning settings, and more via armrest screens, while front occupants can also manage rear display content. At the distributed sharing level, selected systems support desktop file and application sharing across different displays, facilitating cross-device control and two-way content exchange.

Industry Application and Evolution Trends

Multi-screen interaction is swiftly moving from an optional high-end feature to a mainstream cabin standard. A prime example is the SAIC Volkswagen ID.ERA 9X, which features a nine-screen interconnected setup encompassing the instrument cluster, central infotainment screen, front passenger display, rear ceiling monitor, and second-row armrest screens. In this setup, navigation can be mirrored to the cluster and rear screens, while multimedia content can be shared across front and rear rows. Meanwhile, HarmonyOS Cockpit 4.0 leverages the Pangu AI large language model and multimodal sensing upgrades to deliver a more seamless distributed multi-screen experience.

The technology is evolving from sheer "hardware stacking" towards "experience integration". On one hand, screen counts in modern cabins continue to increase, with nine or more linked displays becoming a key competitive benchmark for flagship models. On the other hand, cross-screen content flow is advancing from "unidirectional casting" to "bi-directional collaboration"—allowing rear occupants to not only receive content, but also actively assist with navigation route planning and media selection.

Technical Limitations

The smoothness of multi-screen interaction remains heavily reliant on a unified underlying software architecture. Because display layouts, operating systems, and communication protocols still lack industry-wide standardisation, cross-brand multi-screen interoperability remains difficult to achieve. Additionally, multi-display networking places far greater demands on the graphics rendering capabilities and task scheduling efficiency of the cockpit SoC; inadequate processing power or suboptimal resource allocation can result in screen desynchronisation and interface lag.